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Dear Commission President, Commission Members, and Mr. Rhorer: 

The Office of the Controller’s City Services Auditor Division (CSA) engaged Sjoberg Evashenk 
Consulting, Inc. (SEC) to audit the grant management practices of the Human Services Agency 
(HSA) of the City and County of San Francisco (City) related to Arriba Juntos, a nonprofit, 
community-based organization. Arriba Juntos is contracted by HSA to provide grant program 
services to the City’s residents. 

Reporting Period: July 1, 2014, through June 30, 2016 

Results: 

Arriba Juntos should ensure that its programs consistently meet performance goals, track 
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EXECUTIVE SUMMARY 


Purpose of the Audit 

The City Services Auditor (CSA) hired Sjoberg Evashenk Consulting, Inc., (SEC) to conduct a 
performance audit of the grant management practices of the Human Services Agency (HSA) of 
the City and County of San Francisco (City) related to Arriba Juntos, a nonprofit, community- 
based organization. 
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INTRODUCTION 


The City and County of San Francisco (City) Office of the Controller’s City Services 
Auditor Division (CSA) asked Sjoberg Evashenk Consulting to conduct a performance 
audit of the grant management practices of the City’s Human Services Agency (HSA). 
HSA administers many grant programs, including those associated with addressing 
homelessness, health care, childcare and education, and job training and placement. 
Many of these programs benefit low-income families, children, the elderly, veterans, and 
disabled individuals and are supported with federal, state, and local funding. Arriba 
Juntos is a nonprofit, community-based organization contracted by HSA to provide grant 
program services to the City’s residents. 

SCOPE & OBJECTIVES 

The main objectives of this audit were to: 

• Determine whether Arriba Juntos properly administered the HSA grant in 
accordance with applicable laws, regulations, guidelines, and terms and 
conditions of the grant awards. 

• Assess the adequacy of HSA’s management of Arriba Juntos grant agreements. 

• Assess the effectiveness of Arriba Juntos grant-funded programs. 

The scope of this grant compliance audit covered Fiscal Years 2010-11 through 2014- 
15. 

METHODOLOGY 

To achieve the objectives, the audit tasks and tests performed included: 

• Identifying requirements regarding the administration of HSA’s grant 
management activities. 

• Identifying the universe of grants administered by HSA, understanding the 
purpose and objectives of each grant program, selecting a sample of grants and 
grant recipients to review in depth. 

• For the selected grants/grant recipients: 

o Identified HSA employees involved in grant management activities, their 
roles and responsibilities, and interviewed each. 

o Reviewed the funding amounts, sources, and basis (program costs, 
performance, etc.) for the last five fiscal years. 

o Obtained an expenditure report for each grant and selected a sample of 
expenditures. Obtained documentation associated with the sample of 
expenditures to ensure compliance with the grant agreement. 

o Determined HSA processes to ensure all services funded by the grant 
agreement are effectively and efficiently provided by grantees, including 
performing a walk-through of all related processes and services. 
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o Reviewed related grant performance reports generated by the grantees. 

o Evaluated HSA's ability to measure the performance and outcomes 
associated with the grant agreements and its ability to determine whether 
the outcomes met the objective of the grant. Focused on the most recent 
complete Fiscal Year, 2014-15. 


STATEMENT OF AUDITING STANDARDS 

We conducted this audit in accordance with generally accepted government auditing 
standards. These standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence obtained 
provides a reasonable basis for our findings and conclusions based on our audit 
objectives. 
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AUDIT RESULTS 


Arriba Juntos provides a variety of job readiness training, job placement, and education 
programs through multiple grant agreements with the City. HSA is tasked with the 
oversight and monitoring of each grant agreement program. While our observations of 
Arriba Juntos and HSA identified dedicated and enthusiastic staff, audit testing 
discovered multiple areas of potential program improvements. Program improvements 
include Arriba Juntos consistently meeting program performance goals, tracking 
program outreach and recruitment activities, and maintaining sufficient, organized client 
records. Similarly, areas of improvement related to HSA include increased and 
consistent communication among HSA staff and further developing its oversight and 
monitoring practices to include more in-depth, detailed program assessment activities. 
Specifically, this audit identified the following findings: 

• Progress reports Arriba Juntos submitted indicate several performance goals 
were not met and reported outcome indicators could not always be validated. 

• Arriba Juntos could better document its recruitment efforts associated with 
homeless programs. 

• HSA's annual program monitoring efforts could be more rigorous. 

• Better communication and collaboration among HSA staff could improve program 
efficiency and effectiveness. 


Finding 1: Progress Reports Indicate Several Performance Goals 
Were Not Met and Reported Outcome Indicators Could Not 
Always Be Validated 

The City has several agreements with Arriba Juntos to provide a variety of subsidized 
employment and educational services to San Francisco residents (“clients”). On behalf 
of HSA, Arriba Juntos provides these services through the eight programs listed in Table 
1 . 
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Table 1. Program Descriptions 


Program Name 

Brief Program Description 

2014-15 

Expenditures 1 

1. Individual Referral (IR) 

Job Readiness Training 

$182,333 

2. Community Jobs Program 
(CJP/CJP-1) 

Job Readiness Training & Subsidized 
Temporary Employment 

$1,594,003 

3. Youth Employment Program 
(YES) 

Job Readiness Training 

$278,742 

4. Clean Streets Program 

Job Readiness Training & Subsidized 
Temporary Employment 

$111,078 

5. Interrupt, Predict, and Organize 
(IPO) Program 

Job Readiness Training & Subsidized 
Temporary Employment 

$101,015 

6. Vocational English as a Second 
Language Immersion Program 
(VESL/VIP) 

Job Readiness Training & Subsidized 
Temporary Employment 

$2,235,475 2 

7. Housing and Urban 

Development Homeless 
Employment Collaboration 
(HUD HEC) 

Job Readiness Training 

$181,383 

8. HomeWORC 

Job Readiness Training 

$99,097 


HSA Vocational Assessors meet with CalWORKs or PAES grant recipients (clients) to 
perform an initial evaluation of employability and ability to participate in HSA’s 
employment programs. If the Vocational Assessors determine that a client is ready to 
participate in an employment program, they will make a recommendation to HSA’s 
Employment Specialists regarding the type of employment program or activity that would 
best meet the client’s needs. Employment Specialists formally assign clients to an 
employment program and establish contracts, called employment plans, detailing the 
specific requirements and plans intended to move the client from welfare to work. 
Participation in the employment programs allows clients to meet CalWORKs’ work 
activity requirements. 


Per HSA expenditure reports. 

2 Includes $305,430 in expenditures for “Rapid Response,” a separate effort aimed at helping 
clients with CalWORKs public assistance grants quickly make up missing hours to meet 
requirements. 


5 | P a g e 


SJOBERG EVASHENK 


Each of these programs prescribe specific weekly work and/or job readiness training 
hour requirements for participants that are aimed at increasing the employability of the 
HSA client with an ultimate goal of placement in permanent, unsubsidized jobs. Job 
readiness training typically involves classes designed to improve clients’ preparedness 
for entering the workforce, such as vocational career training, English language skills 
building, modern office skills training, interpersonal skills building, and barrier removal 
activities. Some of these programs also provide subsidized temporary employment work 
that typically involves assignment of clients to a nonprofit host site to gain marketable 
work experience. Some of these programs also help HSA’s clients meet work 
participation requirements (employment hours) associated with their public benefit 
assistance grants. 

With the exception of the Individual Referral program that is paid on a performance 
basis, Arriba Juntos is reimbursed for the costs associated with operating each of the 
programs; largely payments reimburse salaries for Arriba Juntos employees and some 
indirect costs. Arriba Juntos may pass through costs associated with subsidized 
employment wages for clients who are initially paid by Arriba Juntos. Based upon 
proposals submitted by Arriba Juntos for each program, and ultimately the final 
agreements, HSA establishes annual line-item budgets detailing the specific amounts 
that can be reimbursed, including approved positions and salaries. 

Arriba Juntos Progress Reports Indicate Several Performance Goals Were Not Met 

Agreements between HSA and Arriba Juntos for the administration of seven of the eight 
programs reflected in Table 1 include criteria for cost reimbursement and detail specific 
performance goals that must be achieved. Three of the key performance goals that 
generally cross each program relate to enrollment, completion, and placement activities. 
On a monthly basis, Arriba Juntos is required to submit progress reports to HSA 
detailing the progress made toward meeting performance goals. 

We reviewed the performance outcomes submitted by Arriba Juntos for Fiscal Year 
2014-15, which is the most recent complete year, to determine whether key performance 
goals were met. Table 2 reflects a comparison of the key performance goals (per 
contract agreements) and performance outcomes (per Arriba Juntos’s submitted 
progress reports) of each of the seven agreements in Fiscal Year 2014-15. 
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Table 2. Fiscal Year 2014-15 Performance Goals and Outcomes 3 



Enrollments 

Completions 

Placements 

Program 

Name 

Goal 

Outcome 

Goal 

Met? 

Goal 

Outcome 

Goal 

Met? 

Goal 

Outcome 

Goal 

Met? 

HUD HEC 

69 

45 

N 

N/A 

N/A 

N 

37 

22 

N 

HomeWORC 

30 

30 

Y 

24 

22 

N 

11 

17 

Y 

CJP 

CJP-1 4 

160 

250 

150 

254 

N 

Y 

120 

N/A 

101 

N/A 

N 

N/A 

72 

150 

62 

164 

N 

Y 

YES 

160 

62 

N 

96 

46 

N 

30 

40 

Y 

VESL/VIP 

80 

93 

Y 

48 

68 

Y 

41 

31 

N 

IPO 

(Cohort #4) 

N/A 

N/A 

N/A 

10 

7 s 

N 

6 

4 3 

N 

Clean Streets 

125 

93 

N 

100 

28 

N 

28 

19 

N 


Source: June 30, 2015 program progress reports submitted by Arriba Juntos. 


As illustrated in Table 2, Arriba Juntos had 21 enrollment, completion, and placement 
performance goals across the seven programs — “N/A” means that the particular program 
did not have an associated goal outlined in the agreement. While Arriba Juntos 
successfully met both of its performance goals for the CJP-1 program, when comparing 
performance results Arriba Juntos submitted to HSA against all 21 stated goals, we 
noted that Arriba Juntos’s performance outcomes did not meet 67 percent (or 14 of 21) 
of the goals for 2014-15. It is important to note that while Arriba Juntos met its placement 
goal for the YES program, the placement goal was modest to begin with as it is set as a 
percentage of actual enrollments. Because Arriba Juntos did not meet its enrollment 
goal by a large margin, the corresponding target for placements was reduced. 
Furthermore, when looking at performance outcomes during several earlier years, we 
also found that Arriba Juntos similarly did not consistently meet its performance goals. 

Because salaries and wages account for the vast majority of the costs to operate each 
program, we looked to see if staffing levels contributed to Arriba Juntos’s inability to 
meet its performance goals in 2014-15. Specifically, we compared Arriba Juntos’s 2014- 
15 budgeted staff salaries — the salaries of those employees directly providing program 
services, such as case management, instruction, etc. — to the actual amount of staff 
salaries expended. As illustrated in Table 3, we found three programs that spent close to 


3 The Individual Referral grant agreement with Arriba Juntos does not include specific 
performance goals and therefore is not included in this comparison chart. 

4 CJP and CJP-1 have separate performance goals for the same program, but are part of a single 
agreement. 

5 Progress reports not submitted for FY 2014-15. Arriba Juntos provided supplemental 
information. 
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their entire salary budgets, while four programs significantly underspent their salary 
budgets, largely due to vacancies in key staff positions including job developer, case 
manager, instructor, and program coordinator. 

Table 3. Budgeted versus Actual Program Salaries 


Program Name 

Budgeted Salary 

Actual Salaries 

Percent 

Difference 

HUD HEC 

$102,840.00 

$101,516.00 

-1% 

HomeWORC 

$47,572.00 

$46,773.00 

-2% 

CJP (includes CJP1) 

$192,037.00 

$180,495.00 

-6% 

YES 

$146,728 

$131,536 

-10% 

VESL/VIP (includes 
Rapid Response) 

$201,520.00 

$166,740.00 

-17% 

IPO 

$38,405.00 

$27,521.00 

-27% 

Clean Streets 

$93,016.00 

$54,153.00 

-42% 


Interestingly, when comparing reported performance outcomes reflected in Table 2 to 
the salary analysis in Table 3, we were unable to establish a definitive link between 
Arriba Juntos’s staffing levels and its ability to meet program goals. For example, the 
HUD HEC program closely utilized all of its budgeted salaries, but did not meet program 
goals, whereas the VESL/VIP program mostly met program goals while underspending 
its salary budget by 17 percent. 

According to Arriba Juntos, staff vacancies do impact its ability to meet performance 
goals because Arriba Juntos encourages its employees, often previous clients who 
participated in the programs, to take advantage of external employment opportunities 
that can promote their individual professional growth, resulting in increased Arriba 
Juntos staff turnover. Additionally, Arriba Juntos believes that several contract goals 
need to be revised to reflect the current realities of providing program services, citing 
that HSA activities often have a direct impact on Arriba Juntos’s ability to meet 
performance goals. For example, for several of the programs, HSA is responsible for 
referring clients to Arriba Juntos for program enrollment and, as previously described, 
some programs set completion and placement goals as a percentage of the enrollment 
goal. However, if insufficient clients are referred to Arriba Juntos or participants fail to 
enroll in the programs, enrollment goals cannot be met. Consequently, if enrollment 
goals are not met, Arriba Juntos would never have an opportunity to meet completion or 
placement goals. 

Unfortunately, data related to the number of clients referred to Arriba Juntos is not 
tracked by HSA program managers. Thus, we could not determine the number of 
candidates referred to Arriba Juntos, assess the numbers failing to attend Arriba Juntos 
orientation, or determine whether too few referrals caused some of the noted issues 
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related to not meeting performance goals. Tracking and comparing referral data against 
enrollment data would also allow HSA and Arriba Juntos to assess client “fall out” rates 
for each program and determine the reasonableness of performance measures. 

According to HSA, its focus has been on ensuring Arriba Juntos provides quality 
program services and has not concentrated heavily on monitoring whether specific 
performance goals are achieved. HSA stated that now that the programs are more 
established, a greater effort will be made to monitor Arriba Juntos’s progress towards 
meeting performance goals. Additionally, HSA agreed that the program agreements 
should be reviewed and potentially modified to ensure that performance goals align with 
HSA expectations and performance goals can be realistically achieved. 

Outcome Indicators Reported to HSA Could Not Always Be Validated 

As described earlier, Arriba Juntos provides a variety of subsidized employment and job 
readiness training services and is required to achieve specific annual performance 
goals — three key performance goals that generally cross all programs are associated 
with enrollment, completion, and placement activities. Arriba Juntos must report 
performance outcomes linked with these goals to HSA via monthly progress reports. To 
assess the general reliability of performance outcome information submitted to HSA, we 
tested completion and placement outcome data submitted during Fiscal Year 2014-15 
associated with four of the seven programs reflected in Table 2. The four programs 
selected for testing were HUD HEC, HomeWORC, CJP, and Clean Streets. 

Overall, the effort to verify the performance outcomes proved difficult for several 
reasons. First, Arriba Juntos does not maintain a comprehensive listing of the individuals 
included in the performance outcome numbers submitted to HSA. For example, we 
requested a listing of the names of the individuals that were counted in the placement 
performance outcomes associated with the CJP program. Arriba Juntos informed us that 
such a listing is not maintained; rather, to identify the 62 individuals would require us to 
1) review each and every client file of individuals that participated in the program, 2) 
identify the documented outcome within each client file (i.e. , completed program, 
attendance issues, placement, etc.), and 3) create the listing of all clients deemed a 
placement outcome in Fiscal Year 2014-15. Further complicating the process to create 
and verify reported performance outcome figures was the fact that Arriba Juntos client 
files are decentralized, making it difficult for staff to quickly locate all of the necessary 
client files that would be needed for such a review. Although we made initial attempts to 
recreate the placement outcome listing for CJP, this effort could not be reasonably 
accomplished by us. Therefore, we requested Arriba Juntos to take the steps needed to 
provide the completions and placement outcome information. 

Once we obtained the requested completions and placement outcome details from 
Arriba Juntos, we tested a sample of clients reported as being a “completion” or a 
“placement” outcome to verify that documentation available in client files supports the 
outcome reported. For a performance measure to be deemed a “completion” or 
“placement” outcome, the following must occur: 

• Completion — generally, clients must complete a specific number of education 
and/or subsidized work hours for a particular number of months. Arriba Juntos 
must demonstrate that clients complete the required education and/or work hours 
by submitting monthly time and attendance reports to HSA for each client. Clients 
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are also typically deemed a completion if they are placed in unsubsidized 
employment prior to finishing the full length of the program. 

• Placement — generally, clients must be placed in permanent unsubsidized 
employment. Arriba Juntos must demonstrate evidence of the unsubsidized 
employment placements through a copy of a paycheck stub or an employment 
verification letter on business letterhead. 

Completions Outcome Testing 

Arriba Juntos attempted to provide listings of completion outcome information that 
corresponded with data submitted in the monthly progress reports; however, it was not 
always able to provide a complete listing for the four programs selected for review. 
Specifically, while Arriba Juntos was able to provide completions outcome data for the 
HomeWORC and CJP programs, it was unable to identify any clients who would have 
made up the 28 reported completion outcomes related to the Clean Streets program. For 
the two programs where completion outcome information was provided, testing revealed 
one of the programs properly counted each sampled client file as a completion outcome 
- either having met all required hours or having been placed in unsubsidized 
employment prior to finishing the full length of the program - and one did not. The review 
of the HomeWORC program discovered one client who remained in the program through 
to the program end date, but, the client had multiple absences throughout the duration of 
the program. Arriba Juntos could not provide records to demonstrate any of the missed 
hours were made up. The fourth program reviewed did not have performance goals 
associated with completion outcomes. Completion outcome testing results are reflected 
in Table 4. 


Table 4. Completion Testing Results 


Program 

Name 

2014-15 Completion 
Outcomes 

Number of Client 
Files Tested 

Results 

HUD HEC 

N/A- 

no associated goal 

N/A- 

no associated goal 

N/A- 

no associated goal 

HomeWORC 

22 

4 

3 of 4 properly counted as 
a completion. 

CJP 

101 

9 

9 of 9 properly counted as 
a completion. 

Clean Streets 

28 

N/A 

Arriba Juntos was unable 
to identify any client files 
that comprised the 
completion outcome 
figure submitted to HSA. 
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Attainment of Required Participation Hours 

Our audit work also revealed that Arriba Juntos lacks a comprehensive system to ensure 
clients complete the required number of training and/or work hours on an ongoing basis 
while they are still participating in the programs. To document attainment of the required 
participation hours for the training component of the programs, clients sign a daily 
attendance log maintained by Arriba Juntos and to document fulfilment of the required 
work hours, host sites (subsidized employers) submit weekly attendance reports to 
Arriba Juntos detailing the number of hours worked by the clients. Our testing revealed 
the following: 

• HomeWORC — three of four client files tested were missing evidence of meeting 
required program training and/or work hours while the clients were in the 
program. 

• CJP — four of nine client files tested were missing evidence of meeting required 
program training and/or work hours while the clients were in the program. 

According to Arriba Juntos, while it reviews the daily attendance logs to accurately report 
monthly client attendance and absences to HSA, staff does not consistently monitor 
client absences to identify clients needing to make up missed hours and does not 
consistently document when clients have come in to make up missed hours. 

Auditor observations noted instances where clients signed in and out of the attendance 
log at the beginning of the class period, undermining the attendance process to ensure 
participants attend an entire class session. Additionally, when looking through the 
historical attendance logs of one grant program, we observed daily sign-in logs that were 
blank or reflected clients that did not consistently sign in to the log. According to Arriba 
Juntos clients often attend class, but, they are reluctant or unwilling to sign the daily 
attendance logs. Attendees need to be accounted for — either Arriba Juntos must require 
attendees to sign in and out or the course instructor must verify attendance at the 
beginning and end of class. 

We noted that one class instructor incorporated an effective attendance tracking 
practice. The instructor made the sign-in log available only at the very beginning of class 
to sign in and again at the end of class to sign out to ensure students were present the 
entire class period. 

Further, Arriba Juntos practices do not include evaluating the client’s total program 
attendance hours to determine whether the client met the minimum number of required 
hours prior to deeming completion of the program and submitting the outcome to HSA as 
a completion. For example, the VESL/VIP program requires Level 1 clients participate in 
30 hours per week of English language courses for the duration of the six-month 
program. However, Arriba Juntos did not sum up and calculate a client’s total classroom 
attendance hours when determining client completion status. Instead, once the 
appropriate calendar time has elapsed and the client successfully completes the 
applicable, required coursework, Arriba Juntos deems the program element complete 
without fully reviewing actual client attendance hours. 

Program attendance is a significant component within each program agreement. This 
provision is illustrated by contract language requiring Arriba Juntos to ensure its clients 
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meet the specific minimum program participation hours assigned as negotiated with 
HSA. However, by not enforcing attendance log practices that ensure clients attended 
the full class session, not monitoring and following up with clients who need to make up 
hours after absences, and not evaluating actual total program attendance hours for each 
client at program end, clients may not realize the full extent of benefits afforded them, 
and Arriba Juntos may not be ensuring that clients appropriately achieve the standard 
for minimum program hours and program completion status. 

Placements Outcome Testing 

Similar to completions outcome testing, while Arriba Juntos attempted to provide listings 
of placement outcome information to correspond with data submitted in the monthly 
progress reports, it was not always able to provide a complete listing. For example, 
Arriba Juntos successfully identified the clients reported as placement outcomes for the 
HUD HEC, HomeWORC, and CJP programs. However, Arriba Juntos could only identify 
six of the clients who made up the 19 placement outcomes reported for the Clean 
Streets program for the period reviewed. 

Our testing of placement outcomes, as reflected in Table 5, revealed that for two of the 
four programs reviewed, HUD HEC and HomeWORC, Arriba Juntos provided proper 
counts for each program and each client file reviewed showed appropriate evidence of 
an unsubsidized employment placement outcome. 

Our testing relating to the CJP program showed that three of the 50 client files identified 
as unsubsidized employment placements were missing evidence of employment 
verification and one client placed into the Rapid Response program was erroneously 
identified as a placement. In regard to the Rapid Response placement, Arriba Juntos 
indicated that HSA will occasionally approve a placement outcome credit for clients 
removed from CJP by HSA and then placed into the Rapid Response program, which is 
aimed at assisting individuals who are not actively engaged and are missing required 
work participation hours. Nonetheless, placements into this program would not qualify as 
an unsubsidized placement outcome under the definitions in the contract agreement. 
HSA agreed that allowing placements into Rapid Response to count as placement 
outcomes is not in line with the criteria detailed in the grant agreement. 

In regard to the Clean Streets program, while all six client files reviewed included 
evidence of employment verification, the employment verification support for one client 
indicated that the client’s hire date was two days prior to enrollment into the Clean 
Streets program. Since the client obtained employment prior to enrolling in and receiving 
Clean Streets program services, this client should not have qualified as a placement 
credit for the Clean Streets program. 
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Table 5. Placement Testing Results 


Program Name 

2014-15 Placements 
Outcomes 

Number of Client 
Files Tested 

Results 

HUD HEC 

22 

5 

5 of 5 demonstrated 
evidence of placement. 

HomeWORC 

17 

3 

3 of 3 demonstrated 
evidence of placement. 

CJP 

62 

50 

4 of 50 missing evidence 
of applicable 
employment. 

Clean Streets 

19 

6 

1 of 6 missing evidence of 
applicable employment. 


While the audit found that employment verification documentation was not always 
present in client files, it is important to note some of the challenges Arriba Juntos faces 
with obtaining such documentation. Specifically, according to Arriba Juntos, once clients 
obtain permanent employment and exit the programs, clients may not respond to 
communications from Arriba Juntos requesting employment verification support. 
Furthermore, when Arriba Juntos contacts employers directly to obtain the information, 
some employers are unwilling to complete an employment verification letter, or 
employers utilize a service that charges Arriba Juntos a fee for providing employment 
verification information. Nonetheless, even with such obstacles, program grant 
agreements clearly require evidence demonstrating that clients successfully obtain 
permanent employment in the form of a pay stub or employment verification letter. 

We also noted that when Arriba Juntos successfully obtains a paystub for employment 
verification purposes, such documentation does not generally indicate the employee’s 
hire date. Without obtaining evidence of the employee’s hire date, the employment 
verification support may not demonstrate the appropriateness of deeming an individual 
as a placement outcome, such as identifying instances where clients started their 
unsubsidized job before enrollment into one of the grant programs. 


Recommendations 

The Human Services Agency should: 

1. Require Arriba Juntos to track and maintain the names of each client identified as a 
performance outcome on monthly performance reports submitted to HSA for each 
grant program, such as enrollment, completion, and placement outcomes. 

2. Conduct ongoing evaluations comparing reported performance outcomes to 
minimum performance goals detailed in each program agreement to monitor Arriba 
Juntos’s progress towards meeting performance goals. Document and discuss 
results with Arriba Juntos and determine appropriate follow-up. 
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3. On at least an annual basis, obtain a detailed listing of referrals sent to each program 
by applicable HSA Eligibility Workers, Employment Specialists, or other staff 
authorized to refer clients into grant programs. Detailed referral information should 
include the name of the person referred, program referred to, and referral date. Track 
and assess individual program referrals to the individual program goals, calculate fall 
out rates for each program, and evaluate the reasonableness of performance 
measures. Periodically discuss results with Arriba Juntos. 

4. Periodically review program agreements to ensure performance goals align with HSA 
expectations and performance goals may be realistically achieved. Update 
agreements as required to reflect current needs and objectives. 

5. Ensure Arriba Juntos consistently monitors client absences to identify clients needing 
to make up missed hours, develop a mechanism to record and track when clients 
make up missing attendance hours, evaluate each client’s total program attendance 
hours to determine program progress towards meeting minimum hour requirements, 
and incorporate oversight and monitoring practices that reinforce existing attendance 
log procedures to ensure logs are filled out as intended. 

6. Require Arriba Juntos to develop a centralized filing system for program documents. 
Client files, program monitoring and tracking reports, attendance logs, and other 
applicable program documents should be maintained in a central location to prevent 
any changes in program staff from impeding access to current and historical client 
records. 

7. Ensure Arriba Juntos incorporates a process for consistently verifying that sufficient 
documentation that supports reported performance outcomes has been obtained and 
placed in client files. 


Finding 2: Arriba Juntos Could Better Document Its Recruitment 
Efforts Associated With Homeless Programs 

For six of the programs reflected in Table 1, HSA is responsible for determining 
individuals’ eligibility to participate and referring individuals to Arriba Juntos for 
enrollment. Conversely, for two programs, HomeWORC and HUD HEC, which are 
aimed at providing training to the homeless community, individual program eligibility is 
determined by Arriba Juntos as part of its recruitment activities; HSA does not refer 
individuals to these two programs. However, since HomeWORC participants must be 
homeless and receiving public assistance to be eligible to participate, their eligibility 
status is jointly confirmed by HSA (public assistance verification) and Arriba Juntos 
(homelessness certification verification). 

The only training program for which HSA does not have a direct role in determining or 
confirming participation eligibility is the HUD HEC program as eligibility for this program 
is determined solely by Arriba Juntos. As homelessness is the only eligibility criterion, 
program participants are not required to provide any personal identification; rather, 
participants need only provide certain documents determined to be a homeless 
certification that is in line with federal homeless self-certification requirements. While the 
need to demonstrate personal identity is significant in the other programs reviewed, such 
as those in which clients receive wages or public assistance and need to prove their 
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identity for tax or benefit eligibility purposes, the HUD HEC program offers job readiness 
training and job placement services. 

As part of these program services, Arriba Juntos also provides participants with barrier 
removal services, such as helping participants to obtain a State identification card or to 
register for and obtain a copy of their social security card as part of assisting clients in 
job placement. 

Homeless Certification Testing Results 

Since Arriba Juntos is responsible for documenting eligibility for the HUD HEC and 
HomeWORC programs, we reviewed the homeless certification documentation on file for 
participants in these programs. Specifically, each program includes a requirement that 
participants provide homeless certification, which may be in the form of a letter from a 
homeless shelter or a written and signed statement from the client. We identified 45 
HUD HEC and 30 HomeWORC program participants enrolled in FY 2014-15, of which 
we selected 10 HUD HEC and three HomeWORC client files for detailed review of 
homeless certification. In all cases, client files contained appropriate evidence of 
homeless certification through a letter from a homeless shelter or a written and signed 
statement from the client. 

Homeless Programs Recruitment Efforts 

Since Arriba Juntos is solely responsible for determining program eligibility and recruiting 
participants for the HUD HEC program, it is imperative that Arriba Juntos document its 
efforts to reach the program’s target population. According to the HUD HEC contract 
agreement, Arriba Juntos’s outreach efforts should target homeless populations included 
in the following four groups: 

• Veterans 

• Disabled individuals 

• Ex-offenders 

• Parents with children 

According to Arriba Juntos, its outreach efforts include visiting local homeless shelters 
and community events targeted to the homeless population to recruit potential program 
participants. Beginning in 2015, Arriba Juntos began implementing some tracking and 
reporting practices of its outreach activities, including identifying the organization or 
community event attended, the date of event or outreach activity, and the names of 
clients referred resulting from the outreach activities. However, in reviewing the outreach 
tracking spreadsheets and reports maintained by Arriba Juntos, we identified instances 
where this information was not consistently captured and recorded. 

While Arriba Juntos consistently tracks the ethnicity and gender of its participants, it 
does not consistently track and evaluate the types of data that would allow HSA to 
measure Arriba Juntos’s success in reaching the target population. Incorporating a 
formal method for tracking its outreach efforts that is consistently utilized by Arriba 
Juntos staff would allow Arriba Juntos to improve upon its current efforts to fully 
demonstrate and measure its homeless outreach efforts to the target population, as well 
as allow HSA to assess the effectiveness of the outreach efforts. While the other 
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program aimed at helping the homeless community, HomeWORC, does not target 
specific segments of the homeless population (such as veterans, disabled individuals, 
etc.) as the HUD HEC program does, Arriba Juntos is also responsible for recruitment 
efforts for HomeWORC and should consider incorporating similar improvements to its 
recruitment tracking strategies for this program as well. 


Recommendation 

The Human Services Agency should: 

8. Require Arriba Juntos to incorporate a formal method for tracking its outreach efforts, 
including outreach frequency and the names of organizations or community events 
visited, and broaden the demographic information captured from each client to 
include data that would demonstrate outreach efforts made to target populations 
listed in program agreements. 


Finding 3: HSA Annual Program Monitoring Efforts Could Be 
More Rigorous 

HSA program management staff conducts annual monitoring reviews of each contracted 
Arriba Juntos program. The purpose of these reviews is for HSA to determine whether 
Arriba Juntos is delivering the agreed-upon services and to evaluate Arriba Juntos’s 
progress towards meeting annual outcome goal requirements. Key review activities 
include conducting site visits to the Arriba Juntos offices, interviewing Arriba Juntos 
contract management staff, reviewing client files, and assessing performance data. 

According to Arriba Juntos, HSA staff typically spends from a few hours up to one full 
work day on site conducting each program review. In reviewing Programmatic 
Monitoring Reports prepared by HSA staff during Fiscal Year 2014-15 to delineate 
review findings, we found that HSA reviewed five of the eight programs listed in Table 
1 — there were no reports for the Individual Referral, IPO, or Clean Streets programs. For 
the five programs reviewed, the reports indicated that generally HSA found that Arriba 
Juntos effectively provided the services required in the agreements and mostly met or 
was on track with meeting performance outcomes. However, several findings described 
in this audit suggest that HSA’s monitoring efforts could include more rigorous 
examinations of Arriba Juntos program activities. 

For example, HSA relies on Arriba Juntos’s self-reported monthly performance outcome 
information to determine Arriba Juntos’s progress towards meeting its annual 
performance outcome goals. However, HSA does not include in its annual monitoring 
review an independent verification of the underlying support to validate the reported 
outcomes. As noted earlier in this report, auditors attempted to verify Arriba Juntos’s 
reported Fiscal Year 2014-15 completion and placement outcomes, but were generally 
unable to do so due to the lack of available comprehensive information. Had HSA 
attempted similar verification efforts as part of its annual monitoring reviews, it would 
likely have found similar issues with validating outcome data reported by Arriba Juntos. 

Additionally, HSA conducts its assessment based on files selected by Arriba Juntos 
rather than HSA controlling which client files are reviewed. By allowing Arriba Juntos to 
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determine which files are examined, HSA cannot verify the universe of client files or the 
completeness of recordkeeping for the program year under review and may miss the 
opportunity to identify missing files or assess inadequate files. A more rigorous review 
approach would have HSA obtain a list of all program participants (the universe of client 
files for a program), select a sample of files to review from the universe, and request that 
Arriba Juntos provide those specific files to be examined. Moreover, HSA could generate 
a universe of client files from enrollment, completion, and placement outcome 
information associated with monthly progress reports submitted by Arriba Juntos and 
compare the lists. This could occur if recommendations outlined in this report are 
implemented directing Arriba Juntos to maintain complete listings of individuals 
associated with the outcome data in the progress reports. 

Finally, the annual reviews did not appear to incorporate a critical review of program 
facilities, which may provide additional insight into the quality of program services and 
administration. For example, during our walk-through of Arriba Juntos classrooms, we 
identified a risk related to clients undermining the accuracy of attendance sign-in logs by 
signing in and out for the full class day at the beginning of class. While HSA’s monitoring 
efforts may have involved this type of review, the 2014-15 Programmatic Monitoring 
Reports did not address this topic. 

By incorporating more rigorous examinations of Arriba Juntos program activities, 
including validating the accuracy of reported program outcomes and evaluating Arriba 
Juntos’s overall program administration, HSA will be able to better substantiate the 
success of the grant programs administered by Arriba Juntos. HSA should also ensure 
each program is evaluated on an annual basis. 


Recommendation 

The Human Services Agency should: 

9. Develop more rigorous annual program examinations to better substantiate the 
success of each program. Annual program examinations should include validating 
the accuracy of reported program outcomes, evaluating overall program 
administration, and conducting walkthroughs of program facilities. 


Finding 4: Better Communication and Collaboration Among HSA 
Staff Could Improve Program Efficiency and Effectiveness 

HSA has not established a consistent link between the number of clients referred by 
HSA to the employment and training programs administered by Arriba Juntos and the 
budget available to provide program services for most programs. There are four key 
types of HSA staff that are involved with the employment and training programs 
administered by Arriba Juntos and are responsible for the following associated activities: 

• Employment Specialist — Conducts client assessments, establishes client 
specific requirements, and refers clients for enrollment into specific programs. 
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• Program Managers — Liaison between Arriba Juntos and HSA. Primarily tasked 
with oversight and monitoring of Arriba Juntos program activities and 
performance. 

• Contract Staff — Establishes and monitors the annual budget for each Arriba 
Juntos grant program. 

• Fiscal Staff — Receives, reviews, and approves monthly reimbursement invoices 
from Arriba Juntos. 

We found a lack of critical communication among these key HSA staff and inadequate 
collaboration when monitoring the available budget and determining capacity for client 
referrals into programs. For example, with the YES program, Employment Specialists 
refer clients directly to Arriba Juntos to enroll clients into the various programs. However, 
Employment Specialists do not always work with the other HSA staff to ensure sufficient 
program funding is available when referring clients. As a result, in this instance nearly 
the entire Fiscal Year 2015-16 program budget for the YES program was consumed 
midway through the program year causing all new referrals to the program to cease. 
Conversely, the HSA program manager overseeing the Individual Referral program 
requires that Employment Specialists notify and receive signed approval that sufficient 
budget is available before referring clients to Arriba Juntos for enrollment in that 
program. 

Additionally, we found that HSA Program Managers are not involved in reviewing 
monthly invoices submitted by Arriba Juntos. Rather, HSA fiscal staff receives, reviews, 
and approves all reimbursement invoice requests from Arriba Juntos. Although assigned 
fiscal staff conducts line-item comparisons between the approved program budget and 
the reimbursement invoices to determine the allowability of invoice payment, fiscal staff 
does not consult with Program Managers to inquire on Arriba Juntos’s appropriate 
program performance prior to approving invoices for payment. Consequently, invoices 
may be approved for payment when Arriba Juntos has not maintained sufficient progress 
towards its performance outcome goals. 


Recommendations 

The Human Services Agency should: 

10. Ensure that its Employment Specialists communicate with other HSA staff to confirm 
sufficient program funding is available prior to referring clients to various programs. 

11. Incorporate invoice reviews by Program Managers, in addition to Fiscal staff, to 
ensure appropriate performance achievement by Arriba Juntos prior to invoice 
approval and payment. 
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ATTACHMENT: 

DEPARTMENT AND CONTRACTOR RESPONSES 


City and County of San Francisco 


Human Services Agency 



Edwin M. Lee, Mayor 


Department of Human Services 
Department of Aging and Adult Services 


Trent Rhorer, Executive Director 


July 25, 2016 

Tonia Lcdiju, Director of City Audits 
City Hall, Room 476 
1 Dr. Carlton B. Goodlelt Place 
San Francisco, CA 94 1 02 

SUBJECT: SF CSA HSA Arriba Juntos Audit response and Recommendation 
and Response 

Dear Ms. Lediju, 

Thank you for providing us the opportunity to review your findings and respond 
to the recommendations related to the compliance audit of the San Francisco 
Human Services contracts with Arriba Juntos. 

Attached is the requested Recommendation and Response form. SF HSA 
concurs with 9 of the 1 1 Recommendations and partially concurs with 2 of the 
recommendations. HSA is providing a Response for all 1 1 Recommendations and 
has cither implemented, or is in the process of implementing the specific actions 
detailed in the responses. Wc appreciate the thorough review and are satisfied 
with the overall statement: “San Francisco Human Services Agency: Oversight of 
Arriba Juntos Grants Needs Improvement to Better Ensure Delivery of Services.” 

If you have any questions or need additional information, please contact Candace 
Gray at 415-557-6546. 


.QitK'AFAlv 



Daniel B. Kaplan 

Deputy Director for Finance and Administration 

Cc: Trent Rhorer 

Noelle Simmons 
Tony Lugo 
Bill Wedemeyer 
John Tsutakawa 
Candace Gray 
Mashariki Lawson 


P.O. Box 7988, San Francisco, CA 94120-7988 ■ (415) 557-5000 ■ vrtvw.sfhsa.org/ 
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City Services Auditor Division 
Recommendation and Response Form 

Audit Subject: Arriba Juntos 

For each recommendation, the Human Services Agency should indicate whether it concurs, does not concur, or partially concurs. If the Human 
Services Agency concurs with the recommendation, it should indicate the expected implementation date and implementation plan. If the Human 
Services Agency does not concur or partially concurs, it should provide an explanation and an alternate plan of action to address the identified 
issue. 


Recommendation 

Response 

The Human Services Agency should: 

1 . Require Arriba Juntos to track and maintain the names of each client 
identified as a performance outcome on monthly performance 
reports submitted to HSA for each grant program, such as 
enrollment, completion, and placement outcomes. 

HSA concurs. Arriba Juntos has begun collecting and 
tracking monthly individual client data which is represented in 
the monthly reports for all of their contracts beginning on 
7/1/2016. HSA will be sent the individual client detail report 
on a monthly basis with the summary report to review and 
compare with HSA records. 

2. Conduct ongoing evaluations comparing reported performance 
outcomes to minimum performance goals detailed in each program 
agreement to monitor Arriba Juntos’s progress towards meeting 
performance goals. Document and discuss results with Arriba 

Juntos and determine appropriate follow-up. 

HSA concurs: HSA will continue to monitor the monthly 
reports comparing outcomes to performance goals, and will 
meet and discuss with Arriba Juntos on an as-needed basis, 
with a minimum of an annual meeting to review progress in 
meeting annual goals. HSA will meet with Arriba Juntos in 
August, 2016 to begin this process and review and follow up 
on the results of the evaluation. 


City and County of San Francisco, Office of the Controller - City Services Auditor Division 
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City Services Auditor Division 
Recommendation and Response Form 


Recommendation 

Response 

3. On at least an annual basis, obtain a detailed listing of referrals sent 
to each program by applicable HSA Eligibility Workers, Employment 
Specialists, or other staff authorized to refer clients into grant 
programs. Detailed referral information should include the name of 
the person referred, program referred to, and referral date. Track 
and assess individual program referrals to the individual program 
goals, calculate fall out rates for each program, and evaluate the 
reasonableness of performance measures. Periodically discuss 
results with Arriba Juntos. 

HSA concurs: HSA will create a system for tracking referrals 
from HSA into the contracted services more consistently. 

This data will be maintained in the Client Tracking Database. 
HSA will discuss concerns regarding the reasonableness of 
the performance measures with Arriba Juntos, and will 
continue to give monthly feedback, as needed. 

4. Periodically review program agreements to ensure performance 
goals align with HSA expectations and performance goals may be 
realistically achieved. Update agreements as required to reflect 
current needs and objectives. 

HSA concurs. HSA will track and review performance goals 
on a monthly basis, discuss with Arriba Juntos if there are 
concerns about the practicality of the objectives, and modify 
them, as needed, on an annual basis. Begin August, 2016. 

5. Ensure Arriba Juntos consistently monitors client absences to 
identify clients needing to make up missed hours, develop a 
mechanism to record and track when clients make up missing 
attendance hours, evaluate each client’s total program attendance 
hours to determine program progress towards meeting minimum 
hour requirements, and incorporate oversight and monitoring 
practices that reinforce existing attendance log procedures to ensure 
logs are filled out as intended. 

HSA concurs: HSA will monitor monthly each participant’s 
required hours of participation for the month in contracts 
requiring Work Participation hours. Individual case files and 
the attendance log procedures will be reviewed for this 
documentation in the monitoring visits in July/August 2016, 
and ongoing. 

6. Require Arriba Juntos to develop a centralized filing system for 
program documents. Client files, program monitoring and tracking 
reports, attendance logs, and other applicable program documents 
should be maintained in a central location to prevent any changes in 
program staff from impeding access to current and historical client 
records. 

HSA concurs: HSA will require a centralized filing system for 
program documents and consistent organization of both the 
current and historical records for easy access and will review 
progress in the monitoring visits in July/August 2016. 


City and County of San Francisco, Office of the Controller -City Services Auditor Division 
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City Services Auditor Division 
Recommendation and Response Form 


Recommendation 

Response 

7. Ensure Arriba Juntos incorporates a process for consistently 
verifying that sufficient documentation that supports reported 
performance outcomes has been obtained and placed in client files. 

HSA concurs: HSA will conduct a more extensive review of 
the client files in a random selection process in the annual 
reviews beginning August, 2016. 

8. Require Arriba Juntos to incorporate a formal method for tracking its 
outreach efforts, including outreach frequency and the names of 
organizations or community events visited, and broaden the 
demographic information captured from each client to include data 
that would demonstrate outreach efforts made to target populations 
listed in program agreements. 

HSA partially concurs: Arriba Juntos will document the dates 
and events of the organizations and events attended to enroll 
participants. HSA will update the contract language 
specifically for the HEC contract to clarify the service 
population for Arriba Juntos among the 9 partners in the 
contract. 

9. Develop more rigorous annual program examinations to better 
substantiate the success of each program. Annual program 
examinations should include validating the accuracy of reported 
program outcomes, evaluating overall program administration, and 
conducting walkthroughs of program facilities. 

HSA concurs: Formal training for contract staff was 
implemented this year and will include a module on 
monitoring contracts. This issue will also be addressed in the 
quarterly WTW Contracts staff meetings along with a review 
of current monitoring practices, informal training, and 
monitoring shadowing pairing more experienced staff with 
those less experienced. 

10. Ensure that its Employment Specialists communicate with other 

HSA staff to confirm sufficient program funding is available prior to 
referring clients to various programs. 

HSA partially concurs; Employment Specialists are not part of 
this process. The HSA Contract Manager and Program 

Monitor will set up a system for both to monitor the available 
program funding and expenditures to ensure appropriate 
invoicing and expenditure patterns. Begin August, 2016 


City and County of San Francisco, Office of the Controller - City Services Auditor Division 
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City Services Auditor Division 
Recommendation and Response Form 


Recommendation 

Response 

1 1 . incorporate invoice reviews by Program Managers, in addition to 
Fiscal staff, to ensure appropriate performance achievement by 

Arriba Juntos prior to invoice approval and payment. 

HSA concurs: The HSA Program Monitor will review the 
outcomes in the Reporting section of the Contract 
Administration, Reporting and Billing Online (CARBON) 
system to ensure appropriate performance achievement prior 
to approval and payment. This system sends electronic 
notices to both the Contract Manager and Program Monitor 
for the contract when invoices or reports are received each 
month. This will ensure that the two staff work together to 
review and monitor spending trends and funding availability, 
and, as necessary, confirm payment of the invoice. 


City and County of San Francisco, Office of the Controller - City Services Auditor Division 
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ARRIBA 



JUNTOS 


1850 Mission Street • San Francisco, California 94103-3599 • www.arribajuntos.org 
Telephone (415) 487-3240 • Fax (415) 863-9314 • E-Mail info''# arribajuntos.org 


August 1, 2016 


Ms. Tonia Lediju 

Director of the City Audit Division 
City Hall, Room 476 
1 Dr. Carlton B. Goodlett Place 
San Francisco, CA 94 1 02 

Subject: SF- CSA- HSA- Arriba Juntos Audit 


Dear Ms. Lediju, 


In response of your letter dated February 17, 2016. Thank you for giving us the opportunity to 
review your recommendations. 

Arriba Juntos concurs with your recommendations, and will be working with the Human 
Services Agency on such recommendations. Also, AJ will work with the HSA in identifying 
funding to build centralized data base/track clients activities to fulfill this objective. We are 
meeting often to ensure client delivery of services is in compliance. 

If you have any questions, please contact Nenette Tabernilla at 415-551-1966. 


Sincerely, 



Dalila Ahumada 
AJ Executive Director 


An Organization for Business, Education and Community Advancement 
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